External metadata for streaming runner v1 changes by stankiewicz · Pull Request #36373 · apache/beam

stankiewicz · 2025-10-03T12:28:08Z

This change adds:

experiment flag (element_metadata_supported) for dataflow streaming runner
empty proto serialization to windmill sink and empty proto deserialization from windmill readers
plenty of placeholders (todo [Feature Request]: OpenTelemetry Support #33176) where metadata object from proto has to be propagated to windowedValue, starting with cdc, drain flag and trace context

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

gemini-code-assist · 2025-10-03T12:28:42Z

Summary of Changes

Hello @stankiewicz, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request lays the groundwork for integrating external metadata support within the Apache Beam Dataflow streaming runner. The changes introduce an experiment flag to control the activation of this feature, establish the initial serialization and deserialization mechanisms for metadata protos in the Windmill backend, and strategically place placeholders for the subsequent full implementation of metadata propagation throughout the WindowedValue lifecycle. This foundational work is crucial for enabling more sophisticated data processing and observability features in streaming pipelines by allowing additional, user-defined metadata to be carried alongside data elements.

Highlights

External Metadata Experiment Flag: Introduced an experiment flag (element_metadata_supported) to enable external metadata support for the Dataflow streaming runner, allowing controlled rollout of this new feature.
Windmill Sink/Reader Metadata Handling: Implemented foundational serialization of empty metadata protos to the Windmill sink and deserialization from Windmill readers, establishing the basic mechanism for metadata transfer.
Metadata Propagation Placeholders: Added numerous 'TODO' comments across various components (e.g., CDC, drain flag, trace context) to mark future integration points for propagating metadata objects to the WindowedValue structure.
PaneInfo Extension: Extended the PaneInfo class to include a flag indicating whether it contains element metadata, and updated its coder to handle this new information during serialization and deserialization.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2025-10-03T15:07:03Z

Assigning reviewers:

R: @m-trieu for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

stankiewicz · 2025-10-03T15:09:04Z

R:@kennknowles

kennknowles

If I understand correctly this is a noop but it adds the ability for the Windmill shuffle to read/write extended metadata, preparing for adding it end-to-end. Yes?

kennknowles · 2025-10-03T15:43:31Z

                public void processElement(
                    @Element KV<K, ValueInSingleWindow<V>> kv,
                    OutputReceiver<KV<K, V>> outputReceiver) {
+                  // todo #33176 specify additional metadata in the future


Yea I think we'll want to find a way to update Redistribute to automatically carry along "all" metadata. We built a separate data type for Redistribute to avoid making WindowedValue public. But now that it is public and everything around it is public (or public enough) we should make it possible to be resilient to new metadata being added without having to edit Redistribute or the overrides.

suggestions? :)

kennknowles · 2025-10-03T15:45:39Z

+      // element metadata is behind the experiment
+      boolean elementMetadata = WindowedValues.WindowedValueCoder.isMetadataSupported();
+      if (elementMetadata) {
+        PaneInfoCoder.INSTANCE.encode(paneInfo, stream);


IIRC in the design there would be a bit in the PaneInfo that indicates if there is any metadata in the stream, right? Or I guess that is only for the coder and we always put it in the WindmillSink/WindmillSource exchange?

ah, missed that , it should be:
PaneInfoCoder.INSTANCE.encode(paneInfo.withElementMetadata(elementMetadata), stream);

kennknowles · 2025-10-03T15:47:39Z

+
+    Assert.assertEquals(value.getValue(), decodedValue.getValue());
+    Assert.assertEquals(value.getTimestamp(), decodedValue.getTimestamp());
+    Assert.assertArrayEquals(value.getWindows().toArray(), decodedValue.getWindows().toArray());


This doesn't actually have any extended metadata, right?

correct, asserts will have to be added once we have any kind of metadata added, currently this is noop.

but it tests actual serde with metadata (even empty)

kennknowles

Since this is mostly a preparatory noop, fine with me.

kennknowles · 2025-10-07T16:59:42Z

Kicked the tests. I think the particular windmill test that failed may be a known flake - potentially worth deflaking if it hits another time.

stankiewicz · 2025-10-10T12:16:54Z

Run Java PreCommit

stankiewicz · 2025-10-13T13:29:45Z

xlang issue is unrelated.

stankiewicz · 2025-10-13T13:43:17Z

Run Java_IOs_Direct PreCommit

stankiewicz · 2025-10-13T15:00:54Z

Run Java PreCommit

stankiewicz · 2025-10-13T19:29:07Z

Run Xlang_Generated_Transforms PreCommit 3.9

stankiewicz · 2025-10-13T19:34:35Z

Run Xlang_Generated_Transforms PreCommit

stankiewicz added 4 commits October 2, 2025 13:02

encode empty metadata within WindowedValue

4d87d3c

add todos for future otel metadata

7cfd6f8

add todos for future otel metadata

5b7e0fc

spotless

7654478

github-actions Bot added java runners dataflow labels Oct 3, 2025

fix tests

0274977

github-actions Bot added the Next Action: Reviewers label Oct 3, 2025

stankiewicz mentioned this pull request Oct 3, 2025

[Feature Request]: OpenTelemetry Support #33176

Open

17 tasks

kennknowles reviewed Oct 3, 2025

View reviewed changes

stankiewicz added 2 commits October 3, 2025 19:33

fix

bbb9474

fix

99edc76

kennknowles approved these changes Oct 3, 2025

View reviewed changes

stankiewicz mentioned this pull request Oct 10, 2025

[WIP] Drain in element metadata v1 #36471

Closed

3 tasks

stankiewicz added 2 commits October 13, 2025 14:48

fix bugs

1e38bec

fix bugs

a4054cd

liferoad merged commit b9a8972 into apache:master Oct 13, 2025
25 of 28 checks passed

stankiewicz mentioned this pull request Oct 20, 2025

Extending WindowedValue with ElementMetadata #35180

Closed

3 tasks

Uh oh!

Conversation

stankiewicz commented Oct 3, 2025

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist Bot commented Oct 3, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions Bot commented Oct 3, 2025

Uh oh!

stankiewicz commented Oct 3, 2025

Uh oh!

kennknowles left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kennknowles left a comment

Choose a reason for hiding this comment

Uh oh!

kennknowles commented Oct 7, 2025

Uh oh!

stankiewicz commented Oct 10, 2025

Uh oh!

stankiewicz commented Oct 13, 2025

Uh oh!

stankiewicz commented Oct 13, 2025

Uh oh!

stankiewicz commented Oct 13, 2025

Uh oh!

stankiewicz commented Oct 13, 2025

Uh oh!

stankiewicz commented Oct 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants